Papers with multimodal conversational agents
“I’ve Seen Things You People Wouldn’t Believe”: Hallucinating Entities in GuessWhat?! (2021.acl-srw)
Copied to clipboard
| Challenge: | a problem with natural language generation systems is the generation of tokens that are unrelated to the source input. |
| Approach: | They propose two new models to play the GuessWhat?! referential game . they propose to adapt the best visual processing models available to mitigate this issue . |
| Outcome: | The proposed models generate few hallucinations compared to other models available in the literature. |
Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actions (2026.acl-long)
Copied to clipboard
| Challenge: | Recent reinforcement learning (RL) has been widely explored for adapting MCAs to various human-AI interaction scenarios. |
| Approach: | They propose to use a latent action space for reinforcement learning instead of RL to fine-tune MCAs. |
| Outcome: | The proposed method outperforms baselines on two conversation tasks with a novel cycle consistency loss. |
Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodality has been explored in multi-party and multi-session conversations, but task-specific constraints have hindered its seamless integration into dynamic, natural conversations. |
| Approach: | They propose a multimodal conversation dataset and a model with multimodal memory retrieval to equip chatbots with "eyes and ears" they aim to integrate multimodality into chatbot interactions by integrating visual and auditory inputs into the chatbot. |
| Outcome: | The proposed model demonstrates the ability to engage in long-term conversations with multiple speakers in complex, real-world-like settings, effectively processing visual and auditory inputs to understand and respond appropriately. |